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ABSTRACT 

The use of evaluation findings for program 
improvement and decision making was explored with the philosophy that 
different levels of information use occur — from altering perceptions 
of a program, to influencing major decisions about it. The strategy 
of deliberately involving potential information users such as 
clients, sponsors, and other audiences in an evaluation study was 
employed. A brief discussion is presented on the viability of user 
participation as an evaluation strategy, with special attention to 
its potential for increasing the use of evaluation information, A 
number of program changes are outlined that resulted from an 
increased use of evaluation information by the program staff. An 
analysis is presented of why the program staff, in this instance, 
were very receptive the information generated by the evaluation 
study, and why they used it so readily to revise the program. (JD) 



********************************************************************* 

* Reproductions supplied by EDRS are the best that can be made 

* from the original document. 

*******************>************************************************* 



FINDING THE EVALUATION PHILOSOPHER'S STONE, OR 
HOW AN EVALUATION STUDY IS ACTUALLY HELPING PROGRAM IMPROVEMENT 



by 

Joseph J. D'Amlco 
Judith A. Dawson 



"PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) ." 



U.S. DEPARTMENT OF EDUCATION 

NATIONAL INSTITUTE OF EDUCATION 

fcOUt.ATiONAi Rf SOURCES INFORMATION 

f.lNMH 'F RiO 
w This I— .-.♦■♦■t '-.is tppiorfiii hi] as 

iHifivHif tii*- t»* "»»»'■ •>» oiqa»»i/d»«>»» 

V-'i*.» ■ »>.!"•}♦•. * .«/• h»M»»i ni,}fj«* to im(>(OVP 
rp{i<*f<li«t f, »i" - J. 



• P»i—!s .1* vi'A ■.♦ «.i »■*. suu-rt »< !hi4f]f(f U 

itifi'l rn ** ••S-».l»-«y ft»|»»H\»»lil t>Hit ca< Nlfc 
[ifiSitm'- •.* ; y 



Research for Better Schools, Inc. 

444 North Third Street 
Philadelphia, Pennyslvania 19123 



Paper presented at the annual meeting of the American Educational Research 
Association, Chicago, April 1985. 




FINDING THE EVALUATION PHILOSOPHER'S STONE, OR 
HOW AN EVALUATION STUDY IS ACTUALLY HELPING PROGRAM IMPROVEMENT 

The use of evaluation findings for program improvement has been a 
theme in evaluation literature for nearly 20 years. In contrast to their 
early expectations that evaluation studies would provide a significant 
portion of the knowledge base that policy makers would consider when 
undertaking important decisions about a program, evaluators have adopted a 
more realistic view of how much evaluation, in fact, is used and how it is 
used in decision making. They now see that many factors other than formal 
inquiry (for example, practical and political considerations, human biases, 
or common sense knowledge) often strongly influence the directions that 
programs take (Kennedy, 1984; Lindblom & Cohen, 1979; Weiss, 1980). 
Evaluators also now recognize that "use" can be a relative concept and that 
different levels of information use occur — from altering perceptions of a 
program, to influencing major decisions about it (Alkin, Daillak, & White, 
1979). And they have identified several strategies for enhancing knowledge 
use, one of which is to deliberately involve potential information users 
such as clients, sponsors, and other audiences in evaluation studies (for 
example, Gold, 1983; Stake, 1975). 

In this paper we shall take a closer look at the information use issue 
ae we describe how one evaluation study, employing user participation, has 
enhanced knowledge utilization and, in fact, has influenced program 
decisions, We begin with a brief discussion of the viability of user 
participation as an evaluation strategy with special attention to its 
potential for increasing the use of evrluation information. We illustrate 
this discussion with a description of our own evaluation study and our 
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experiences involving program staff in it. We then outline a number of 
program changes that we feel resulted from an increased use of evaluation 
information by program strff . Finally, in an analysis of this situation, 
we seek to answer the questions: "Why were program staff so receptive to 
the information generated by the evaluation study? 11 and "Why did they use 
it so readily to revise the programs? 11 

Knowledge Use in Evaluation 

As Ernest House noted, "Producing data is one thing: getting it used 
is quite another 11 (1973, p. 133). This sentiment describes a major, 
chronic dilemma of evaluators. As far back as the middle f 60s both Guba 
(1968) and Stufflebeam (1967) noted that evaluation information is often 
seen as useless and irrelevant to decision makers. Since then knowledge 
use — or non use — - has been documented quite diligently by evaluators and 
policy researchers (for example, Alkin, Daillak, & White, 1979; Deshpande & 
Zaltman, 1983; Florio, Behrman, & Goltz, 1979; Leviton, & Hughes, 1981; 
Weiss & Bucuvalas, 1980) • This attention has improved our understanding of 
the problem of information non use, its causes, and how it might be 
overcome. One strategy that a number of authors suggest to increase 
knowledge use is to involve clients or others in evaluation activities. 

In examining the impact of health evaluation research, Patton and his 
colleagues (Patton, Grimes, Guthrie, Brennan, French, & Blyth, 1975) 
discovered that evaluation information probably does get used by decision 
makers. They suggest, however, that it is only one of a number of pieces 
of information that influence decisions and it often competes with other 
considerations, which Patton and his colleagues label "political" and 



ERLC 



"personal." These authors see the personal considerations as the most 
interesting for they seem to be more susceptible to evaluator influence. 
These personal considerations include general low commitment and lack of 
enthusiasm for using evaluation information and low regard for the 
evaluator or evaluation design. 

In reviewing the research of Patton and his colleagues as well as many 
others. Leviton and Hughes (1981) describe clusters of variables associated 
with these personal considerations that they feel could enhance a client's 
use of information. These clusters are: relevance of the information for 
program concerns, effectiveness of communication, ease with which the 
clients process the information, credibility of the information, and a key 
actor '8 (client, manager, or developer) involvement and advocacy for using 
the information. 

More recently, Kennedy (1984) and McLaughlin (1984) have zeroed in on 
similar considerations noting that potential users may reconceptualize, 
incorrectly interpret, or even disregard evaluation information if it is 
not expressed in their terms or linked to their experiences. Both authors 
also stress the importance of making potential users comfortable with 
evaluation information as well as the importance of making that 
information relevant. 

Leviton and Hughes (1981) suggest that client involvement in research 
studies is an effective way to make users comfortable with evaluation 
information, while making it relevant for them, and thereby enhancing their 
use of the information. They argue that by involving clients in study 
activities, evaluators stand a better chance of relating research 
information to us*r experiences, needs, and concerns; of making sure the 
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information is communicated smoothly and efficiently; of helping users 
recognize its usefulness; of keeping their own and the evaluation's 
credibility high; and of gaining the advocacy and involvement of a key 
actor. 

Client involvement in evaluation and policy research studies is not a 
unique idea. Two evaluation approaches, one described by S:ake (1975) and 
one by Gold (1983) * explicitly call for user involvement at several stages 
of the evaluation process. Stake's approach, which he labels "responsive 11 
evaluation, calls for the evaluator to consult users and try to incorporate 
their interests and values into the study design. He also recommends that 
their reactions to report drafts be solicited and, whenever possible, they 
be consulted in the interim. Gold goes even farther in his "stakeholder 11 
approach. According to him, the evaluator should adhere to user 
(stakeholder) preferences — in effect, work for the stakeholders, who 
specify what kinds of information they want and in what form. Client 
participation also has been advocated by Ballard and James (1983) and 
Leitko and Peterson (1982), The former suggest several participation 
strategies including using advisory committees, circulating report drafts, 
interacting verbally and in writing, and disseminating research products. 
Leitko and Peterson go beyond Ballard and James by advocating that clients 
actually be involved in research decision making. 

Given the findings and recommendations of these authors, it seems that 
Involving potent Lai users in evaluation activities might be an effective 
way to overcome some barriers to knowledge use. First, it might help clients 
understand research activities more fully than if they were not involved. 
Second, it might make them more aware of the relevance, credibility, and 



value of the evaluation information — particularly if they helped generate 
it. Third, it might improve their communication with evaluators. Fourth, 
it might cause them to become advocates of the evaluation process. Fifth, 
and most important, it might enable the clients to process the information 
more quickly and easily and use it more effectively in decision making. 

We have had the opportunity to test some of these hypotheses and 
suggestions about client involvement in the course of our documentation and 
evaluation of two school improvement programs, School Effectiveness 
Training and the Secondary School Development Program, Both are 
cooperative efforts involving staff from Research for Better Schools, Inc. 
(RBS), from local and state teachers 1 associations, and from schools and 
school districts in Pennsylvania and New Jersey, The clients we have 
involved in our study of these programs are RBS staff who developed the 
programs and provide technical assistance to schools implementing them. 

The SET/SSDP Programs 
School Effectiveness Training (SET) and the Secondary School Develop- 
ment Program (SSDP) are designed to help urban school staff and administra- 
tors establish processes that will enable them to work collaboratively 
toward organizational improvement in their schools. The programs, which 
were developed by RBS and the New Jersey Education Association (NJEA) , are 
based on the premise that schools can become more effective by making 
better use of their human resources, by opening up the decision-making 
process to nil staff members, and by focusing their improvement efforts on 
.i irw i iitii'al aroas at a time. The developers 1 goals for schools that 
embark upon the SET or SSDP processes include higher staff morale, improved 



communication, better working relationships among teaching strff and 
between teachers and administrators, improved learning envi' ... and 
finally, increased academic achievement among students* Throu b .i these: 
programs, schools form coordinating councils that include representatives 
of the teaching, administrative, and support staffs. These councils, 
similar to labor-management councils in private industry, work through task 
groups to identify priority areas for school improvement and to develop — 
and later implement — - plans for addressing these areas. 

Both programs include an evaluation component, staffed by RBS 
evaluators. The study's major mission is to provide information that RBS 
and the NJEA can use to improve the SET/SSDF programs and to explain the 
programs 1 operations to others. This paper focuses on the evaluation 
study's program improvement function which — as will be seen — has 
resulted in modifications to the programs that apply both as the programs 
are introduced in new sites and as adjustments in on-going sites. 

The Evaluation Study 
The evaluation study of SET and SSDP is designed to maximize its 
utility for program improvement. The design is guided by two general 
perspectives. The first emphasizes knowledge use. The evaluators 
recognize that lack of utilization is a pervasive problem in program 
evaluation. Furthermore, they believe that one criterion of an evaluation 
study's effectiveness is its utility. Unless study findings are used, they 
cannot contribute to program improvement. Two particular suggestions from 
the literature on utilization outlined earlier are incorporated into the 
study design. They are the reliance on qualitative methods of inquiry and 



the involvement of program development and technical assistance staff in 
evaluation study activities. 

The second design perspective is that of planned change. The SET and 
SSDP programs involve introducing changes in schools. The evaluators 
believe that by incorporating a planned change perspective the study will 
provide the most useful information for program improvement. Therefore , 
the study includes variables identified by previous researchers as influ- 
encing the process of developing and implementing new educational programs 
(for example, Corbett, Dawson, & Firestone, 1984; Fullan, 1982; Lehming & 
Kane, 1981). 

The evaluation study of SET and SSDP is primarily a qualitative one. 
Data are collected through semi-structured interviewing and observation. 
Program and school staff are interviewed several times each year. In 
addition, evaluators observe program-related activities, which include 
orientation and training sessions, coordinating council meetings, 
discussions between RBS and district staff, inservice sessions, and RBS 
planning meetings. During each data collection activity, evaluators are 
guided by broad lists of questions or categories of information. Data are 
later recorded as field notes. Other evaluation methods include analysis 
of information from surveys, program and school documents (program 
materials, meeting minutes, and newsletters, for example), and school 
records (test scores, student and teacher attendance rates, and the like). 

The program staff who are involved in the evaluation study are RBS 
employees who helped design the SET and SSDP programs and who continue to 
modify and refine them, who provide training for teachers and 
administrators, and who provide technical assistance to help smooth program 
implementation. 



Some program staff are involved as direct participants in the study. 
During most of the evaluation study f s three years, one or two program staff 
members have been assigned to it as active, direct participants. 
Initially, two field agents — whose primary program roles were to help 
develop materials and provide technical assistance to school-level 
participants ~- helped with data collection. They accompanied the 
evaluator to sites, interviewed teachers, and prepared written field notes. 
Later, another staff member (the senior author) became a full participant 
in iU study, contributing to design, data collection, and feedback 
activities. 

Beyond this direct participation, other program staff members (includ- 
ing the director) are involved less directly but nonetheless deliberately 
and extensively in the study. Most of this involvement is related to data 
analysis or feedback activities. These activities involve staff in the 
evaluation study in that they stimulate and encourage two-way communication 
and interaction between the evaluators and the staff members. The 
involvement techniques are; 

Debriefing sessions — These are of two types: (1) After 
visiting a particular site, evaluators usually give informal 
feedback specific to the site visit to the staff members 
responsible for program operations in that site. (2) 
Evaluators ask staff members to describe events the evaluators 
do no** attend as well as other personal or telephone contacts 
with sites. 

Joint site visits — Program staff and evaluators often attend 
onsite program development sessions or other project meetings 
together and compare notes about what happened. 

Field notes — Field notes, which are edited to protect 
informant anonymity, are sometimes shared and discussed with 
individual staff members. 

Informal interaction — Evaluators and program staff sometimes 
exchange general information about sites during brief, 
spontaneous interactions (over lunch, for example) in their 
workplace. 
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Feedback meetings — Evaluation findings relevant to all sites 
and their program implications are discussed during formal 
meetings convened specifically to present recent information. 
Findings also are discussed during routine staff or other 
meetings. 

Written statements — After completing a round of interviews or 
otherwise collecting data from all sites, evaluators sometimes 
prepare written memos presenting tentative findings* 

Formal reports — Formal evaluation reports are written 
annually and distributed to staff members in draft form* 

Potential Risks 

Although we have found these staff involvement strategies have many 
benefits for us as evaluators — including increased use of study findings, 
as will be discussed later — they also have potential risks* So far, in 
contrast to the benefits, the risks are relatively hypothetical. They are 
disadvantages which we think could have occurred but whose negative impacts 
are so slight thus far that we prefer to label them "risks. 11 

One major risk is cooptation of evaluators or developers. For 
instance, evaluators may avoid threatening issues, equivocate negative 
reports, or generally be less than candid in order to protect their rela- 
tionships with developer participants. Program development staff, on the 
other hand, may rely too heavily on information they feel they have helped 
generate — for example, by accepting and acting on evaluators 1 analyses or 
recommendations without reviewing them critically. To a minor degree some 
cooptation of this sort may be occurring in this study, but our impression 
is that to date there have been no serious consequences. 

Another related risk has to do with the integrity of the study. When 
evaluators emphasize collecting information that is useful to ongoing 
program operations, they may neglect background information (whose 
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immediate utility is not apparent) which could become important to under- 
standing how a program evolves in a particular school. That has occurred 
some in this study, but wa feel it has resulted as much from the lack of 
sufficient resources for the study as it has from the study's focus on 
utility. Furthermore % the evaluators are aware of the importance of 
background information and attempt to collect it at every opportunity. So, 
this has not been a serious problem — although it remains to be seen if 
the program might have benefited had more comprehensive background 
information been collected sooner. 

Another, more serious, risk to a study 1 s Integrity is that program 
development staff who participate actively may replace evaluation 
personnel. This may influence the nature and quality of evaluation 
activities. Program developers often have relatively little training in 
evaluation methods. Also, their status may be higher or lower than that of 
evaluation personnel they replaced. Consequently, technical aspects of 
evaluation activities may be deemphasized, tedious tasks — such as those 
associated with compiling quantitative data — may be eliminated, and the 
quality of data collection and analysis may suffer. To some extent, this 
study has been influenced by the level of evaluation expertise and the 
status of participating program development staff members. Quantitative 
activities have been neglected. Some field notes have been cryptic. A 
report draft contained statements that went beyond the data. For the most 
part, however, the evaluation study has proved sufficiently flexible to 
adapt to the type of assistance available, for instance by reducing the use 
of quantitative information. And so far, this direction has been 
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compatible with the staff's information needs* Also helping to minimize 
this risk is one direct participant's training in anthropology and 
consequent understanding of the goals and techniques of qualitative methods 
of inquiry* 

Another risk of involving program staff in evaluation activities is 
the imposition that it makes on their time and priorities. In this instance, 
staff involvement in evaluation takes time away from their developing or 
assisting in the field tasks as they learn new skills or conduct evaluation 
activities* It also means wearing two hats and, in some cases, reconciling 
value conflicts — whether to approach a situation as an evaluator or a 
developer or a field ass is tor. So far, no one has complained too loudly 
about these impositions* More extensive program staff involvement and the 
extra responf Abilities it would bring with i. r*uld likely cause a change, 
however* 
Benefits 

The participation of program staff in the evaluation study has 
benefited the study itself, the program and its staff, and program clients 
at the district and school site level* One of the major benefits, the one 
which is the topic of this paper, has been an increase in the use of 
information generated from the study. Other benefits, which will be 
discussed first, are that costs have been reduced and that the study itself 
has been enriched. 

It appears that evaluation study costs have been reduced because 
additional data are generated through staff participation without the 
expense of hiring extra evaluation personnel, although estimating piecise 
savings is a complex task* There have been some costs associated with 
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direct participation — in ~^hich program 8taff conduct evaluation 
activities euch as interviewing teachers or observing meetings — because 
other work has been displaced and the salaries of program staff are 
probably higher than those of staff who would otherwise be hired to assist 
with evaluation activities. Those costs seem to be offset by the program 
director's assignment of lower priority to the work that was displaced 
than to the evaluation study activities, and by the avoidance of the 
start-up, orientation and training costs of hiring someone new. Indirect 
participation has not required enough time of program development staff to 
displace other work, so it has expanded the data base at virtually no 
expense . 

Staff participation has enriched the evaluation study for several 
reasons. First, of course, it has simply enlarged the data base. Second, 
characteristics of the people involved — namely, the backgrounds and 
perspectives of the program staff members — have enhanced their 
contributions to the study. Two of the direct participants have been 
involved in many previous, similar efforts to develop and implement new 
programs in schools. They understand the process of change and are quick 
to identify factors that facilitate or hinder program-related changes in 
individual sites. A third has helped develop program procedures and 
materials, is very familiar with both programs, and knows what questions to 
ask about them. The staff members who participate less directly also have 
varied perspectives — in organizational development and adolescent 
development, for example — and these perspectives have helped enrich the 
study. Third, the way in which the staff participation strategy is being 
enacted — primarily its interactive quality — has enriched the study. As 
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program staff and evaluators have discussed their observations, 
experiences, and reactions the portraits of the programs and of their 
operationalization in individual schools have become fuller and more 
complex. 

Information Use and Program Changes 
During the evaluation study's three years, we have observed program 
development and assistance staff introduce many modifications. Moreover, 
we believe the evaluation study has influenced many of those changes, 
although its contribution cannot always be distinguished from that of other 
sources of influence. For example, the frequent informal Interaction 
sometimes makes it difficult for staff to recall what information they 
obtained independently, from each other, or from the evaluators. Also, 
program and evaluation staff may both contribute information that allows 
them to piece together a description or interpretation of events in a site. 
And, many program changes have been made for multiple reasons, some that 
were related to the evaluation study and some that were not. 

Despite these difficulties, we can point to several indications that 
the evaluation study has contributed to program improvement. First, some 
information that led to program changes was available only through the 
study. Other information that led to changes emerged as themes that 
recurred during formal and informal evaluation feedback. Finally, 
evaluators were sometimes present when program modifications were made or 
incorporated into onsite activities; we observed staff use evaluation 
information that had been communicated to them recently. 
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was the recognition that schools needed flexibility in identifying issues 
to be pursued through the program. Specifically, initial conceptual links 
to Edmonds 1 (1981) factors of school effectiveness proved to be too 
limiting to program implementation. 
Operational Changes 

These conceptual changes led to numerous operational changes. Field 
agents began to spend more time in sites, providing more follow-up 
assistance. They sought to ensure that their involvement would extend over 
a several-year period. They developed special material* to instruct school 
staff in program procedures and program-related skills. They held 
cross-site seminars where they provided additional technical assistance. 
They introduced annual intensive training sessions for coordinating 
councils. Developers also began to work more closely with administrators, 
corsulting with them during the first program adoption sessions. They 
continued to elicit administrators 1 support and advised them to provide 
specific types of assistance such as time for planning and formal approval 
of council recommendations. In an attempt to introduce flexibility into 
the process of selecting issues, developers urged schools to establish 
sub-groups beyond ones determined by Edmonds 1 five factors, and, in fact, 
to move from standing "factor" groups to temporary "task" groups. 

In sum, evaluation information helped cause program developers to 
modify their initial conceptualization of the programs along several lines. 
This i in turn, led to changes in the way program staff assisted site 
personnel and changes in the way the programs were carried out in the 
schools. 
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An Evaluation Philosopher's Stone? 

Mythology tells us that alchemists once searched for a philosopher's 
stone: a mythical ingredient which, when combined with ordinary metal , 
caused the metal to turn to gold. Have we found the evaluation 
philosopher's stone? Have we found the mythical ingredient which causes 
program developers 9 ordinary knowledge to react with evaluation information 
to produce program improvements? To some extent, but some special 
circumstances and conditions came together for us: 

o the nature of the evaluation study; 

o the nature of the program's underlying philosophy; 
and 

o the nature of the program development staff and 

how they approach their responsibilities. 

The Evaluation Stu dy 

As we have pointed out above, we selected an evaluation strategy that 
involves program developers and assistors because we believe such 
involvement would make it more likely that the developers would relate to 
and use the evaluation information. Souie program staff have participated 
directly in data collection and analysis. Others have been involved 
through extensive, interactive feedback activities. In our opinion, this 
approach has produced the desired results. 

Program developers view evaluation information favorably and they 
consider it reliable often because they helped collect and analyze it. 
They use it to make decisions related to program revisions because it is 
presented to them quickly, informally, and in forms (memos, discussions, 
and conversations) that enable them to relate the evaluation findings to 
their experience';, responsibilities, and plans. 
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Involving program staff in data collection and analysis and communi- 
cating evaluation information to them quickly , informally, and continuously 
also has eliminated evaluation "surprises 11 to some extent. Program 
development staff rarely are shocked by written reports of implementation 
or other problems because they have been exposed to the data already — 
and, in fact, perhaps because they helped develop the analysis — and 
because the reports usually come quickly enough to allow developers to 
adjust the programs or their activities to prevent serious damage at a 
particular site. 

Lastly, in choosing to involve development staff in evaluation activi- 
ties we chose evaluation orientations and approaches to help make evaluation 
information "user friendly." For instance, when we decided to solicit 
program developers 9 input in our selection of categories of data, we did so 
to make sure they would feel comfortable that our information was related 
to their programmatic interests and priorities. Likewise, we selected a 
largely qualitative approach to data collection to help these developers 
feel more comfortable with the data than they would have with data 
generated quantitatively. Our decision to couple evaluation information 
with suggestions for possible courses of action also was chosen to make 
developers comfortable with the evaluation data and to make it easy for 
them to use. Even if our recommendations are not taken, we feel, including 
them will make it more likely that these developers will come to decisions 
about what to do than if only data were presented. Including all these 
considerations seems to have paid off in that program developers have used 
the evaluation information for the program decisions noted auove. 
Moreover, the use of these provisions to make evaluation information easier 
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for developers to digest and use has helped us* It encourages developers 
to focus on a constructive activity — considering alternatives for 
improvement rather than on a non-constructive one — thinking up reasons 
why our evaluation data were wrong. 

Program Philosophy 

Both SET and SSDP have, as a philosophical cornerstone, the notion 
that the progreas should reflect the needs of clients wherever possible* 
Of course this does not mean the programs are totally responsive; but, 
within theoretical boundaries, there is a considerable amount of content 
and process flexibility* This flexibility enables developers to tailor 
aspects of the programs to site needs and situations, even as these change 
over time* This, in turn, gives developers the flexibility to apply 
evaluation data quickly, as it is needed. And, they can apply it 
differently in different sites. Moreover, developers working directly with 
sites are allowed sufficient latitude to make site-specific changes in 
program content and process without having to clear them first. This 
flexibility and onsite discretionary power has often interacted with those 
attributes of the evaluation approach that made the information readily 
available and useable. Data — perhaps collected or analyzed by a 
developer — have been presented informally, its relevance for a particular 
site ascertained — often rather quickly — and modifications identified 
and carried out* 

In addition to this philosophical emphasis on flexibility and respon- 
siveness, there is a clear understanding among developers and evaluators 
that SET and SSDP are pilot programs — for the time being, anyway. This 
understanding has further increased the latitude of program developers to 
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make process or content changes and has contributed to their willingness to 
use evaluation information to make those changes. 

Program Development Staff 

Unique staff attributes and attitudes toward their program responsi- 
bilities and roles have also contributed to their willingness to use 
evaluation information. As noted above, for example, developer 
responsibilities and roles are defined broadly and allow them considerable 
latitude for adjusting the programs to meet site needs. In addition to 
designing program content and processes, developers also serve as field 
agents — introducing the programs and assisting site participants with 
implementation — and, in some cases, as data collectors. In short, they 
are jacks-of-all-trades whose main responsibility is to make sure the 
programs work. This broad role definition has not only made developers 
comfortable with using evaluation information, it frequently causes them to 
seek it out in an effort to understand sites better. It has made them more 
likely to use evaluation information to modify the programs or their own 
activities. And, as noted above, the fact that the programs were 
considered pilot efforts made developers even more receptive to evaluation 
information and even more willing to use it. 

Lastly, the program developers have a considerable amount of sophisti- 
cation regarding evaluation and its value for improvement. Some have been 
evaluators and all have worked with evaluators on previous programs. Their 
backgrounds and experiences give them an understanding of evaluation. They 
are aware of its utility and are not reluctant to use it. 
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Conclusion 



Although our strategy of actively involving program development staff 
in evaluation activities has been a kind of philosopher's stone, we think 
that we have benefited from a combination of circumstances also. The 
nature of the programs we are examining and of the program staff 
have helped the alchemy along. So in sum, we cannot call our strategy a 
panacea for the use of evaluation data dilemma. 

We can say, however, that the involvement of program staff has been 
very beneficial. Beyond helping to get the information listened to, 
believed, and used, the strategy has reduced our costs, enriched our data 
base, and broadened our data's perspective. But perhaps most importantly, 
our efforts to involve program staff in evaluation activities has enabled 
evaluators and developers to come to a shared understanding about how 
educational change programs can be made more effective. So we — and we 
think the program developers, too — recommend that this strategy be 
considered for other evaluation studies. Although the circumstances of 
other evaluators may not be as favorable as ours, the experiences reported 
here suggest that involving program staff to whatever extent possible is 
likely to increase information use. 
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